{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2508ffb0",
   "metadata": {},
   "source": [
    "#### Display a question, answer choices and gold-standard answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Question: Animals may fight, make threatening sounds, and act aggressively '\n",
      " 'toward members of the same species. These behaviors usually occur as the '\n",
      " 'result of')\n",
      "'Answer Options: competition,  conservation,  decomposition,  pollution'\n",
      "'Gold Answer: competition'\n"
     ]
    }
   ],
   "source": [
    "# Extract from prepared dataset\n",
    "from pprint import pprint\n",
    "pprint(\"Question: \"+ worldtree_10[\"worldtree\"][\"train\"][1][\"question\"])\n",
    "pprint(\"Answer Options: \" + \",  \".join(worldtree_10[\"worldtree\"][\"train\"][1][\"choices\"]))\n",
    "pprint(\"Gold Answer: \"+ \"\".join(worldtree_10[\"worldtree\"][\"train\"][1][\"answer\"]))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2508ffb0",
   "metadata": {},
   "source": [
    "#### Display model-generated chain-of-thought and extracted answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Aggressive behaviors are often the result of competition. Competition is the '\n",
      " 'result of animals fighting, making threatening sounds, and acting '\n",
      " 'aggressively toward members of the same species. So, the final answer is A.')\n",
      "'A.'\n"
     ]
    }
   ],
   "source": [
    "pprint(worldtree_10[\"worldtree\"][\"train\"][1][\"generated_cot\"][0][\"cot\"])\n",
    "pprint(worldtree_10[\"worldtree\"][\"train\"][1][\"generated_cot\"][0]['answers'][0]['answer'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The answer generated by the model was correct! To evaluate model answers automatically, ThoughtSource has an in-built evaluate function."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Including full text prompts in your json file (optional, can at any time be recreated from the prompt fragments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d17f5366954d4a1b8543ca01eabeebd5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10 [00:00<?, ?ex/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# If you want to see your prompt as full text, you can use:\n",
    "# This is not done by default as the json file gets very large, choose if you want to save it. You can always reconstruct it later.\n",
    "worldtree_10.full_text_prompts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer the following question through step-by-step reasoning.\n",
      "\n",
      "Animals may fight, make threatening sounds, and act aggressively toward members of the same species. These behaviors usually occur as the result of\n",
      "A) competition\n",
      "B) conservation\n",
      "C) decomposition\n",
      "D) pollution\n",
      "\n",
      "Answer: Let's think step by step.\n"
     ]
    }
   ],
   "source": [
    "# this is the exact text, which was used by the model to generate the chain of thought\n",
    "print(worldtree_10[\"worldtree\"][\"train\"][1][\"generated_cot\"][0][\"prompt_text\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer the following question through step-by-step reasoning.\n",
      "\n",
      "Animals may fight, make threatening sounds, and act aggressively toward members of the same species. These behaviors usually occur as the result of\n",
      "A) competition\n",
      "B) conservation\n",
      "C) decomposition\n",
      "D) pollution\n",
      "\n",
      "Answer: Let's think step by step.Aggressive behaviors are often the result of competition. Competition is the result of animals fighting, making threatening sounds, and acting aggressively toward members of the same species. So, the final answer is A.\n",
      "Therefore, among A through D, the answer is\n"
     ]
    }
   ],
   "source": [
    "# this is the exact text, which was used by the model to generate the chain of thought\n",
    "print(worldtree_10[\"worldtree\"][\"train\"][1][\"generated_cot\"][0][\"answers\"][0][\"answer_extraction_text\"])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "ccbfa654f25866afe66a1c016d0b518a994fbe20a93c2d8b432dbe114020e2d2"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
